{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seldon Core Serving Example\n",
    "\n",
    "Seldon Core comes installed with Kubeflow. The [Seldon Core documentation site](https://docs.seldon.io/projects/seldon-core/en/latest/) provides full documentation for running Seldon Core inference.\n",
    "\n",
    "Seldon Core is an open source platform to deploy your machine learning models on Kubernetes at massive scale.It converts ML models (Tensorflow, Pytorch, H2o, etc.) or language wrappers (Python, Java, etc.) into production REST/GRPC microservices.\n",
    "\n",
    "Seldon handles scaling to thousands of production machine learning models and provides advanced ML capabilities out of the box including Advanced Metrics, Request Logging, Explainers, Outlier Detectors, A/B Tests, Canaries and more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Label current namespace so you can run inference tasks in it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/var/run/secrets/kubernetes.io/serviceaccount/namespace', 'r') as f:\n",
    "    namespace = f.readline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl label namespace $namespace serving.kubeflow.org/inferenceservice=enabled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Create an Istio gateway in current namespace named kubeflow-gateway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -n $namespace -f ./serving/seldon_gateway.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Execute all the below commands in your local environment\n",
    "\n",
    "### 3. Create an example SeldonDeployment with a dummy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -n $namespace -f ./serving/seldon.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.Wait for state to become available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get seldondeployment seldon-model -n $namespace -o jsonpath='{.status.state}\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Port forward to the Istio gateway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8004:80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Send a prediction request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0]]}}' -X POST http://localhost:8004/seldon/$namespace/seldon-model/api/v1.0/predictions -H \"Content-Type: application/json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be able to see a response like below:\n",
    "\n",
    "```\n",
    "{\n",
    "  \"meta\": {\n",
    "    \"puid\": \"i2e1i8nq3lnttadd5i14gtu11j\",\n",
    "    \"tags\": {\n",
    "    },\n",
    "    \"routing\": {\n",
    "    },\n",
    "    \"requestPath\": {\n",
    "      \"classifier\": \"seldonio/mock_classifier_rest:1.3\"\n",
    "    },\n",
    "    \"metrics\": []\n",
    "  },\n",
    "  \"data\": {\n",
    "    \"names\": [\"proba\"],\n",
    "    \"ndarray\": [[0.43782349911420193]]\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Canary Rollout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -n $namespace -f seldon_canary.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for state to become available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get seldondeployment seldon-model -n $namespace -o jsonpath='{.status.state}\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run bash script to send 100 times prediction request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "for i in {0..100};\n",
    "do\n",
    "  curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0]]}}' -X POST http://localhost:8004/seldon/$namespace/seldon-model/api/v1.0/predictions -H \"Content-Type: application/json\";\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the traffic is split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "default_count=$(kubectl logs $(kubectl get pods -n $namespace -l version=example -o jsonpath='{.items[0].metadata.name}') seldon-container-engine -n $namespace | grep \"Calling grpc for transform-input\" | wc -l)\n",
    "canary_count=$(kubectl logs $(kubectl get pods -n $namespace -l version=canary-example -o jsonpath='{.items[0].metadata.name}') seldon-container-engine -n $namespace | grep \"Calling grpc for transform-input\" | wc -l)\n",
    "\n",
    "echo \"\\nThe count of traffic routes to default: $default_count\"\n",
    "echo \"The count of traffic routes to canary: $canary_count\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete -n $namespace seldondeployment seldon-model\n",
    "!kubectl delete -n $namespace gateway kubeflow-gateway"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
